fix: pin app compose commands so an app op can never take down the core stack#161
Merged
Conversation
Compose v2 discovers both the compose file and the project name from the working directory, walking up the tree when cwd has no compose file. App dirs live under the core dir, so an app command run with cwd=installed_apps/<app> and a missing compose file resolves to /core/docker-compose.yml with project name "core" — the live core stack. app_compose_command builds an explicitly pinned command (-f, -p, --project-directory), refuses to run when the app's compose file is missing, and guards against a resolved project name of "core". The project name mirrors compose's own normalization of the directory name so existing app stacks keep their identity.
All app compose invocations relied on cwd for file and project discovery. When an app's docker-compose.yml is missing (e.g. a malformed custom-app zip), compose walked up to the core compose file and ran against project "core" — an app stop/down then stopped shard_core itself, taking the shard offline. Route every app compose call (including the pause-time `ps -q` lookup in memory_pressure) through app_compose_command, which pins -f/-p and refuses to run at all when the app's compose file is absent.
Covers issue #160's acceptance: with installed_apps/<app>/docker-compose.yml deleted, every app operation errors on the missing file and runs no compose command at all, so the core project is never touched. Verified to fail against the pre-fix cwd-based calls (7 failures). The project-name normalization cases were checked against `docker compose config` (v5.0.2) so pinned -p keeps matching the project compose derived from the app dir name for already-installed apps.
reclaim_compose_stack now goes through app_compose_command, which refuses to run without the app's compose file. The test called it for an app dir that never existed — a state unreachable in production, since reclaim only runs right after a successful `compose pause`. Create the compose file the test implicitly assumed.
This was referenced Jul 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #160.
Problem
Every app compose invocation was unpinned —
docker compose <cmd>with onlycwd, no-f, no-p. Compose v2 derives both the compose file and the project name from the working directory and walks up the tree whencwdhas no compose file. App dirs live under the core dir:So an app command run from
installed_apps/<app>with a missing compose file resolved to/core/docker-compose.yml, project namecore— andstop/downthen stoppedshard_coreitself through the mounted docker socket. That is the root cause of the recent full-shard outage: core stopped itself while uninstalling a malformed custom app.Fix
app_compose_command(app_dir)inshard_core/util/subprocess.pybuilds a command pinned with-f <app_dir>/docker-compose.yml -p <project> --project-directory <app_dir>.ComposeFileNotFound) — compose never gets the chance to walk up.ComposeProjectNotAllowedwhen the resolved project would becore(or is empty).up --no-start,up -d,down,stop,pause,unpauseinapp_tools.py, plus the pause-timeps -qlookup inmemory_pressure.reclaim_compose_stack.Project name compatibility
The pinned
-pmust keep matching the project name compose previously derived from the app dir name, or already-installed app stacks would be orphaned.normalize_project_namemirrors compose's own normalization (lowercase, drop everything outside[a-z0-9_-], strip leading_/-), verified empirically againstdocker compose config(v5.0.2):My_App.v2-x→my_appv2-x,_-Foo→foo. This matters because custom-app names come from the uploaded zip filename and can contain uppercase or dots.Blast radius of the new error
Failing loudly is the point, and the existing callers already contain it:
_uninstall_appand_reinstall_appwrap stop/shutdown intry/except Exception(logged), and the installation worker loop catches every exception per task. So uninstalling a malformed app now logs the error, removes the app dir and DB row, and leaves core running — instead of taking the shard offline.docker_stop_all_apps/docker_shutdown_all_appsgather withreturn_exceptions=True.Tests
New
tests/test_app_compose_pinning.py(16 tests, unit tier — no Docker):installed_apps/<app>/docker-compose.ymldeleted, each ofdocker_create_app_containers,docker_start_app,docker_pause_app,docker_unpause_app,docker_stop_app,docker_shutdown_appraisesComposeFileNotFoundand runs no compose command at all (subprocessnever called), so the core project cannot be touched.-fand-pfor the app.coreproject rejected, project-name normalization cases.Verified to have teeth: checked out the pre-fix
app_tools.py/memory_pressure.pyand re-ran the file — 7 failed, 9 passed. With the fix: 16 passed.One existing test needed fixing, and the reason is the guard doing its job:
test_reclaim_compose_stack_reclaims_each_containercalledreclaim_compose_stackfor an app dir that never existed, which now raisesComposeFileNotFound. That state is unreachable in production — reclaim only runs right after a successfulcompose pause— so the test now creates the compose file it implicitly assumed.Note:
docker_start_app's@throttle(5)is global across apps and tests — a start triggered by an earlier test would silently drop the call under test and fail it spuriously, so the throttle clock is reset in an autouse fixture.Local run:
2 failed, 188 passed— the memory_pressure one above (now fixed) andtest_app_lifecycle.py::test_app_starts_and_stops, which fails only on this dev box because another container already binds host port 80 (Bind for 0.0.0.0:80 failed: port is already allocated); it passed on CI's clean runner. CI's first run:1 failed, 189 passed— only the memory_pressure test, fixed in the last commit.Recommended reading order
shard_core/util/subprocess.py— the pinned-command helper and its guardsshard_core/service/app_tools.py— all app compose calls routed through itshard_core/service/memory_pressure.py— the one remaining cwd-based callagents.mdtests/test_app_compose_pinning.pytests/test_memory_pressure.py